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This paper provides a summary of conclusions from the Small Aircraft Transportation 
System (SATS) Higher Volume Operations (HVO) Flight Experiment which NASA 
conducted to determine pilot acceptability of the HVO concept for normal conditions. The 
SATS HVO concept improves efficiency at non-towered, non-radar airports in Instrument 
Meteorological Conditions (IMC) while achieving a level of safety equal to today’s system. 
Reported are results from flight experiment data that indicate that the SATS HVO concept 
is viable. The success of the SATS HVO concept is based on acceptable pilot workload, 
performance, and subjective criteria when compared to the procedural control operations in 
use today at non-towered, non-radar controlled airfields in IMC. The HVO Flight 
Experiment, flown on NASA’s Cirrus SR22, used a subset of the HVO Simulation 
Experiment scenarios and evaluation pilots in order to validate the simulation experiment 
results. HVO and Baseline (today’s system) scenarios flown included: single aircraft 
arriving for a GPS non-precision approach; aircraft arriving for the approach with multiple 
traffic aircraft; and aircraft arriving for the approach with multiple traffic aircraft and then 
conducting a missed approach. Results reveal that all twelve low-time instrument-rated 
pilots preferred SATS HVO when compared to current procedural separation operations. 
These pilots also flew the HVO procedures safely and proficiently without additional 
workload in comparison to today’s system (Baseline). Detailed results of pilot flight 
technical error, and their subjective assessments of workload and situation awareness are 
presented in this paper. 


I. Introduction 

In the United States, the current National Airspace System (NAS) of hub and spoke operations has served its 
purpose well, but it is beginning to reach a capacity plateau. Due to increasing demand on the system and with only 
modest potential gains in the number of flights, the system will reach gridlock within the next 10-15 years. 1,2 
Several new, small, efficient aircraft are being developed by Honda, Cessna, Diamond, Eclipse, Safire, Cirrus, 
Lancair, Adam Aircraft, and others to provide point-to-point service and make use of small airports, many without 
control towers that lie outside Air Traffic Control (ATC) radar coverage. 

When instrument meteorological conditions (IMC) restricts operations to Instrument Flight Rules (IFR) at non- 
towered, non-radar airports, ATC uses procedural separation that restricts operations to only one approaching or 
departing aircraft at a time - the “one-in/one-out” paradigm which severely limits the operational throughput at 
these airports. Air charter operators might be compelled to use these airfields if the IMC operational efficiency can 
be improved. The Small Aircraft Transportation System (SATS) Project breaks the one-in/one-out paradigm and 
expands capacity by allowing multiple, simultaneous operations while achieving a level of safety equal to today’s 
system (Baseline). The concept of operations (CONOPS) that achieves this goal is termed SATS “Higher Volume 
Operations” (HVO). 3 

Several human-in-the-loop experiments were conducted with low-time instrument-rated pilots to address if these 
pilots could fly SATS HVO safely and proficiently and with acceptable workload and situation awareness. And 
based on a positive pilot acceptability of SATS HVO through these experiments, pilots will be able to fly SATS 
HVO with similar, if not lower workload, flight technical error (FTE), and better situation and navigation awareness 
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when compared to the one-in/one-out procedures in use today. Key to this HVO validation process were the results 
from the HVO Flight Experiment described here. 

II. Background 

A. SATS HVO CONOPS Overview 

The SATS HVO concept is based on a distributed decision-making environment that assumes major decision- 
making responsibility for the pilot, and resource mitigation for ground-based automation, the airport management 
module (AMM). The concept utilizes a newly defined flight operations area called a Self-Controlled Area (SCA), 
established during periods of IMC around “SATS designated airports” (i.e., non-towered, non-radar airports). 
Within the SCA, pilots, using new procedures and airborne information systems would have the ability and 
responsibility to maintain separation and spacing between themselves and other similarly equipped airplanes, while 
sharing information with the AMM that in turn sequences approach traffic. 

Aircraft will approach a SATS airport on an IFR clearance granted by ATC to a transition fix above the SCA. 
This fix is also an initial approach fix (IAF) for an instrument approach procedure^ Prior to reaching the fix, the 
pilot requests a landing assignment from the AMM through their onboard system. The AMM responds with the 
SCA entry procedure (standby, vertical or lateral), relative sequence information (follow <Callsign>), and missed 
approach hold fix assignment (MAHF, e.g. ANNIE or CATHY). The AMM only sequences arrivals (including 
missed approach aircraft), not departures. Nominally, up to four arriving aircraft are allowed in the SCA before 
denying entry (issuing a “standby”), though this constraint can be affected by local airspace restrictions . Following 
their entry assignments and the HVO procedure to “descend to lowest available altitude,” pilots are deconflicted 
from other arriving aircraft (i.e., the AMM reserves slots at one of the MAHFs for all aircraft at the IAFs). 



Many of the features of the GPS-T based SATS HVO concept are depicted in Figure 1. SATS arrivals (Red and 
Blue aircraft) to the IAFs with alternating missed approaches, and departures (Green and Purple aircraft) to the 
Departure Fixes (DFs): 

Blue - entering the SCA having coordinated descent with ATC when no other aircraft assigned to CATHY, 
missed approach is blue dashed path, AMM returned: 

LATERAL entry, follow NONE, missed approach CATHY 
Red - having arrived by ATC instruction to transition fix above SCA at 4000ft with one other CATHY 
assignment, the AMM returned: Vertical entry , follow BLUE, missed approach ANNIE 
Purple - departing SCA via departure procedure and contact ATC prior to DF 

Green - released by ATC to depart (within departure window); holding short and using on-board tools to find 
open slot in arrival stream to take the runway and depart 

Pilots that are given a “standby” sequence can track the number of aircraft in the SCA to estimate their delay as 
they continue to their clearance limit, the transition fix at an altitude above the SCA, and hold. When the pilot 

§ GPS-T instrument approach procedures were chosen as a basis for this concept, although other instrument 
approach procedures could be used. 

The number of arriving aircraft, including those executing a missed approach, are limited by the holding altitudes 
available for the approach. Figure 1 shows the nominal approach design with four potential holding segments. 


2 

American Institute of Aeronautics and Astronautics 


receives an AMM entry message with sequence and missed approach information, the pilot checks for an available 
holding altitude, and will request descent from ATC. The pilot can then determine if further descent is prudent by 
following the “lowest available altitude” procedure at the IAF, (clearing for traffic below is the pilot’s self- 
separation responsibility in the SC A). Pilots initiate their approach once adequate spacing behind the lead aircraft 
has been met (determined through either a generic rule-based spacing procedure, i.e., safe for all combinations of 
aircraft performance, or by using an on-board self-spacing tool). The AMM reserves a holding slot for assigned 
missed approaches. A pilot executing a missed approach would climb to the “lowest available altitude” at their 
assigned MAHF and would then be sent a new arrival sequence. 

For SATS departures, pilots will file flight plans with a SATS departure procedure to a departure fix (DF, i.e., 
Figure 1 ELLEN or GINNY). Just as in today’s non-radar environment, the pilot should expect a clearance void 
time and potentially a release time restriction as part of their IFR clearance. This affords seamless integration with 
today’s instrument flight operations. Within this ATC departure window, they will use on-board information/tools 
to deconflict themselves with landing traffic, e.g., ensure no arriving aircraft within 5nm of the airport. The pilot 
would then depart and contact ATC according to the departure procedure before entering ATC controlled airspace. 

B. Validating the SATS HVO Concept and the Use of Piloted Studies 

The SATS HVO research team developed and applied a four phase process to design and validate the HVO 
concept of operations (CONOPS), depicted in Figure 2. 



2. Computer Model 
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1. Concept Model 
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3. HITL Concept 
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Experiment; 
ATC Simulation 
(Summer-Winter 


4. Results 
Validation 






Figure 2. HVO Validation 

Phase one involved HVO CONOPS development. The key safety properties of a draft HVO CONOPS were 
established by mathematical verification methods based on formal logic and theorem proving. 3 4 This study began 
formally verifying that self- separation can be maintained when pilots adhere to the HVO procedures (including 
AMM logic). 

Phase two involved the development of computer simulations. The AMM function and associated algorithms 
were verified and validated using a representative set of HVO scenarios. 

Phase three, providing the primary material of this paper, determined if HVO functionality and procedures were 
safe, fly able and acceptable to pilots. Human-in- the-loop scenarios were developed that compared the SATS HVO 
CONOPS to the one-in-one-out procedural control environment available today (Baseline). 

Low-time instrument rated evaluation pilots (EPs) provided experimental data and subjective feedback as they 
flew the scenarios in experiments using progressively higher fidelity simulation, from a medium fidelity general 
aviation (GA) computer simulation 5 " 6 to the Cirrus SR22 aircraft in flight. Consistent early results across the various 
experiment platforms, including the high-fidelity HVO Flight Experiment, provided confidence in the simulation 
environment, so later studies of the HVO concept were conducted with the simulation platform. 7 Subsequently, two 
additional simulation experiments determined if non-normal procedures were acceptable to the pilot as well as the 
procedure support automation developed by NASA researchers. 8 ' 9 Also, an ATC simulation study was completed 
and focused on determining controller acceptability of the concept model. 10 
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Phase four was a proof-of-concept public demonstration of six SATSLab aircraft flying the SATS HVO 
CONOPS procedures in the 2005 SATS Technology Demonstration held in Danville, Virginia. All four phases 
provided feedback to the improvement of the SATS HVO CONOPS and ultimately toward recommending a viable 
way to improve upon the one-in/one-out procedure in place in the NAS today. 

C. The Need for the HVO Flight Experiment 

Because researchers used an iterative process to increase technology readiness level through HVO research and 
development, the Flight Experiment was a critical step to validate pilot acceptability of a high fidelity HVO system 
in an actual flight environment. Described in the report, “Flight Technical Error Analysis of the Small Aircraft 
Transportation System Higher Volume Operations (SATS HVO) Simulation and Flight Experiments,” 7 an analysis 
of flight technical error (FTE) from the HVO Flight Experiment also validated the HVO Simulation Experiment and 
subsequently the simulation environment, the SATS Air Traffic Operations Lab (ATOL) as a platform worthy of 
conducting further detailed HVO analyses. Also, developing and implementing HVO systems and procedures for 
use in the Cirrus SR22 became a lead-in for a strong NASA leadership of the phase four, full-architecture, multi- 
aircraft, proof-of-concept demonstration. 

III. Experiment Description 

A. HVO Validation Experiment - Objectives and Methodology 

Determining pilot acceptability of HVO meant investigating research objectives to address: 

Comparing the SATS HVO CONOPS to the one-in-one-out procedural control environment available today 
(Baseline)... 

• Can pilots safely and proficiently fly the airplane while performing SATS HVO procedures? 

• Do pilots perceive that workload, while using HVO procedures and tools, is no greater than flying in 
today’s system? 

The following hypotheses were developed to meet the research objectives: 

Flight Technical Error (Baseline & SATS scenarios) 

• Subject pilots will fly within the FAA’s Practical Test Standards (PTS) for the instrument rating 
100% of the time during all scenarios 

• Deviations from assigned flight paths (i.e., RMSE values) will be equivalent across all scenarios 
Subjective Workload (Baseline & SATS scenarios) 

• Equivalent workload ratings will be associated with all scenarios 
Situation Awareness (Baseline & SATS scenarios) 

• Equivalent situation awareness ratings will be associated with all scenarios 

B. Experimental Design 

The experiment design matrix shown in Figure 3 includes the five scenarios flown in the HVO Simulation 
Experiment. 6 Three of these scenarios were repeated in the flight experiment. For the simulation experiment the 
experiment design used for data collection was a 2 (Procedure Type) x 5 (Scenario Type), within-subject design in 
which the same 15 participants (i.e., low-time instrument rated pilots) were assigned to each experimental cell (i.e., 
test condition). Simulation evaluation pilots 1 through 15 (S i_i 5 ) were asked to perform all 10 test conditions in 
partially counterbalanced order under simulated IMC. For the flight experiment, a 2 (Procedure Type) x 3 (Scenario 
Type), within-subject design was used for data collection and 12 of the same 15 evaluation pilots (F i_i 2 ) performed 
six test conditions twice in partially counterbalanced order under simulated IMC. Dependent measures included 
pilot FTE and subjective assessments of workload and situation awareness. 
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PROCEDURE TYPE 



Baseline 

SATS 


Departure 

§ 1-15 

Si-15 

SCENARIO 

TYPE 

Approach Without Traffic 

Si -15 F 1-12 

Si-15 F 1-12 

Approach With Virtual Traffic 

Si -15 F 1-12 

Si -15 F 1-12 


Approach to Missed 
Approach with Virtual Traffic 

Si -15 F 1-12 

Si -15 F 1-12 


Approach with Piloted Traffic 
(Linked Simulation) 

Si -15 

Si-15 


S=Simulation Subjects 
F= Flight Subjects 


Figure 3. Experiment Design Matrix 


C. Scenarios and Profiles 

The HVO experiments were designed to represent the HVO CONOPS through a sample set of scenarios: 

Departure with approaching traffic - this scenario was flown in the simulation experiment only (not reported 
in this paper). 5 6 

Approach without traffic (no holding required). The pilot task was to descend and fly the approach via ATC 
clearance during the Baseline scenarios. During the SATS scenarios, the pilot requested and received a “lateral 
entry, follow none” sequence from the AMM, was handed off into the SCA from ATC, and self-initiated the 
approach. Typically the durations of the SATS scenarios were equivalent to those of the Baseline scenarios. 

Approach with virtual traffic (holding required). This scenario clearly differentiated Baseline from HVO 
efficiency. In Baseline, the pilot waited behind two other aircraft in holding until they had landed and ATC 
provided clearance to begin the approach (i.e., 30+ minutes in holding). In HVO, the pilot followed the AMM 
sequencing behind the two other aircraft, self-separated from the other aircraft in the SCA, and self-initiated the 
approach by following advisories provided by a self-spacing software tool. The SATS scenarios’ durations were 
about half that of the Baseline scenarios. 

Approach to missed approach with virtual traffic (holding required). This scenario included having the pilot 
fly the missed approach. SATS scenarios required flying the missed approach while self-separating in the SCA. 
The SATS scenarios’ durations were about two-thirds as long as the Baseline scenarios’ durations. 

Multi-pilot linked simulation approaches (holding required). This scenario was flown with four linked 
simulators (not reported in this paper). 6 

The experimental scenario flight profiles were based on GPS-T instrument approach procedures. Figure 4 
depicts the progressive development of the experiment flight profile that began with the template SCA in the 
CONOPS and was implemented as shown in the simulation and flight approach charts. Evaluation pilots used these 
to fly experiment scenarios (Baseline charts not shown). The approach chart is a key element of the pilot procedures 
and coupled with task training, all evaluation pilots were able to fly the HVO procedures (described earlier with 
Figure 1) and the accompanying non-precision GPS-T instrument flight approach procedure. 
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Figure 4. HVO Procedure Development Progression -- Concept to Flight Experiment 

D. Progressing from Simulation-to-Flight: Pilot Criteria, Training, and Apparatus 

In choosing a subject pool for the validation experiments, researchers identified criteria for evaluation pilots who 
were capable, but not overly experienced. Although not tested, it was postulated that if the low-time instrument 
rated subject pool validated the concept positively, then more experienced pilots would do the same. Subject pilot 
criteria included low-time (less than 1000 flight hours), an instrument rating, and legal currency to fly IFR. None of 
the participants had previously flown a Cirrus SR22 aircraft, worked as a flight crewmember for an air carrier within 
the last year, or flown for the military. 

The Project’s HVO validation process meant that the HVO flight experiment was flown to validate pilot 
acceptability of a subset of the HVO simulation scenarios. Twelve subject pilots were randomly drawn from the 
original fifteen EPs that participated in the HVO Simulation Experiment. This reduced training requirements for the 
HVO Flight Experiment and allowed pilots to progress logically from hand-flying a medium fidelity general 
aviation (GA) simulator to the Cirrus SR22 aircraft. In the post experiment debriefing, all twelve pilots were 
comfortable that the pre-experiment training they received was adequate to fly experimental scenarios. 

Figure 5 shows the GA simulator and the Cirrus SR22’s instrument panel used for the experiments. Common 
software across platforms was developed to drive the MFD and the Horizontal Situation Indicator (HSI) displays. 
Variation between the simulation and flight profiles was deliberately minimized so as not to alter the experiment 
objectives or hypotheses. Pilots were tasked to manually fly the scenarios in simulated IMC using traditional round- 
dial instruments for primary flight guidance information (i.e., without autopilot). Pilots were tasked to meet FAA 
instrument rating practical test standards criteria during all scenarios (both SATS and Baseline). 11 
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Simulation Experiment Platform 
GA Simulator, similar to Cirrus SR22 


Flight Experiment Platform 
Cirrus SR22 


Figure 5. HVO Experiment Platform Progression (note commonality of HSI and MFD) 


Figure 6 shows the NASA Cirrus SR22 used in the HVO Flight Experiment. On-board were the research 
software components that enabled evaluation pilots to fly a subset of the HVO simulation scenarios. The research 
MFD and HSI were the primary interface devices which pilots used while they hand-flew the SR22 through the 
HVO procedures. The HSI provided course and course deviation information that corresponded to the same GPS 
non-precision approach procedure being flown. The MFD had a moving map with virtual scenario traffic depicted. 
The MFD also provided HVO procedure support through the “pilot advisor” which advised pilots sequentially 
through HVO rules and procedures. Self-spacing software was used and provided support for the initiation of 
approaches. The MFD contained Conflict Detection and Alerting (CD A) capability, but only normal HVO 
scenarios, i.e., without conflicts, were flown. Simulated virtual aircraft and ATC audio was fed into the pilot’s 
headset to build a realistic HVO scenario environment. All HVO systems were on-board for the Flight Experiment, 
and were an interim configuration that formed the foundation for the phase four full-architecture, multi-aircraft 
proof-of-concept demonstration conducted at Danville Regional Airport in June 2005. 







• Research MFD 

• Research HSI 

• AMM, Traffic 
Generator, and 
scenario control 


Figure 6. SATS HVO Cirrus SR22 with Flight Experiment Apparatus 


A large screen capture of the research MFD is shown in figure 7. The information presented on the MFD was of 
significant concern in the workload and situation awareness evaluations. The HVO Simulation Experiment also 
reported usability results of the interface. 
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Figure 7. Research MFD (white own-ship, double-cyan chevron = traffic to follow, green AMM “Entry” 
message with sequence, solid line instrument approach, dotted line missed approach, HVO Pilot Advisor with 

“OPEN APPROACH”) 


IV. Analysis of Results 

Data were collected both during and at the end of the scenarios as well as the end of each session. Quantitative 
results to follow include FTE of airspeed, altitude, and lateral path deviation. Qualitative results shown are from 
workload assessments and situation awareness assessments including traffic and navigation guidance awareness. 

A. Flight Technical Error 

Root Mean Squared Error (RMSE) with respect to Airspeed Target on the approach 

Pilots were instructed to fly the same target airspeeds for both the SATS and Baseline scenarios. Target 
airspeeds were chosen for the three segments of the GPS-T non-precision instrument approach procedure: initial 
(IAF to Intermediate Fix or IF); intermediate (IF to Final Approach Fix or FAF), and final (FAF to Missed 
Approach Point or MAP). The target airspeeds for the initial, intermediate and final segments were 120, 110, and 
100 knots, respectively for the experiment. Instrument rating PTS requires the pilot maintain +/- 10 knots of the 
target airspeed. 1 1 

A significant difference was found to exist between the airspeed target RMSE values associated with each 
procedure type (p = 0.003). A RMSE airspeed deviation of 5.82 knots occurred during the Baseline scenarios 
(Mean = M = 5.82, Standard Deviation = SD = 1.81, Sample Number = N = 72), and a RMSE airspeed deviation of 
4.82 knots occurred during the SATS scenarios (M = 4.82, SD = 1.29, N = 72). 
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Figure 8. Flight Experiment Deviation from Airspeed Target 

Airspeed FTE conclusion: The hypothesis that evaluation pilots would “fly within the FAA’s PTS for the 
instrument rating 100% of the time during all scenarios” was supported by the airspeed target RMSE values. Also, 
figure 8 indicates that evaluation pilots maintained airspeed with respect to an assigned target value more accurately 
when they performed the SATS scenarios than when they performed the Baseline scenarios (second hypothesis was 
supported). Based on the limited context of these findings, pilots can fly SATS HVO procedures safely and 
proficiently when compared to today’s system (Baseline). 


Percent time within Altitude Envelope on the Approach 

Pilots were instructed to fly within the same PTS altitude envelope for both the Baseline and SATS scenarios: 
-100 ft of “at or above” altitudes, and +100 and -Oft for the Minimum Descent Altitude (MDA) until MAP or visual 
transition to landing. 1 1 

No significant difference was found to exist between the percentages of time within altitude envelope associated 
with each procedure type (p = 0.300). During the Baseline scenarios, evaluation pilots flew within the defined 
altitude envelope 97.42% of the time (M = 97.42, SD = 5.44, N = 72). During the SATS scenarios, evaluation pilots 
flew within the defined altitude envelope 98.41% of the time (M = 98.41, SD = 3.58, N = 72). 
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Figure 9. Flight Experiment Percent Time in Altitude Envelope 

Altitude FTE Conclusion: The first FTE hypothesis that subject pilots would “fly within the FAA’s PTS for 
the instrument rating 100% of the time during all scenarios” was not supported, but this does not diminish the 
acceptability of the SATS HVO concept because the second hypothesis that “deviations would be equivalent across 
all scenarios” was supported. Figure 9 indicates that evaluation pilots maintained altitude within an assigned 
envelope equally well when they performed the Baseline scenarios and when they performed the SATS scenarios. 
Based on the limited context of these findings, pilots can fly SATS HVO procedures safely and proficiently when 
compared to today’s system (Baseline). 
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RMSE with respect to Lateral Path Target on the approach 

Pilots were instructed to use target path as approach flight path during both the Baseline and SATS scenarios. 
Instrument PTS is to maintain within % scale deflection of the course deviation indicator (CDI). 11 Full scale 
deflection on the research HSI represented .3 nautical miles through the approach procedure. 

In the flight experiment, a significant difference was found to exist between the lateral path deviation target 
RMSE values associated with each procedure type (p = 0.045). A RMSE lateral path deviation of 0.12 nm occurred 
during the Baseline scenarios (M = 0.12, SD = 0.17, N = 72), and a RMSE lateral path deviation of 0.08 nm 
occurred during the SATS scenarios (M = 0.08, SD = 0.05, N = 72). 
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Figure 10. Flight Experiment Deviation from Lateral Path Target 

Lateral Path FTE Conclusion: The first hypothesis that pilots would “fly within the FAA’s PTS for the 
instrument rating 100% of the time during all scenarios” was supported as was the second hypothesis that 
“deviations would be at least equivalent across all scenarios.” Figure 10 indicates that evaluation pilots maintained 
lateral path deviation with respect to an assigned target value more accurately when they performed the SATS 
scenarios than when they performed the Baseline scenarios. Based on the limited context of these findings, pilots 
can fly SATS HVO procedures safely and proficiently when compared to today’s system (Baseline). 

B. Subjective Assessments of Workload 

Evaluation Pilots (EPs) used the Modified Cooper-Harper Rating Scale to rate the level of workload that they 
experienced during each of the experiment’s six test conditions. Workload ratings could range on a scale from “1” 
(i.e., the instructed task was very easy/highly desirable; operator mental effort was minimal; and desired 
performance was easily attainable) to “10” (i.e., the instructed task was impossible; it could not be accomplished 
reliably). 12 Since EPs performed each of the six test conditions twice, each EP provided 12 workload ratings. For 
each EP, the two workload ratings associated with a given test condition were averaged together to yield a set of six 
workload ratings. As reported below, nonparametric tests were employed as a conservative method for analyzing 
workload ratings associated with discrete rating scale items. 

Procedure Type x Scenario Type 

EPs reported experiencing the following workload ratings when they performed different types of scenarios 
using Baseline procedures and SATS procedures (N=12 for each scenario): 

• Scenario 3 (Baseline, Approach without Traffic): M = 2.08, SD = 0.51; 

• Scenario 4 (SATS, Approach without Traffic): M = 1.46, SD = 0.45; 

• Scenario 5 (Baseline, Approach with Virtual Traffic): M = 2.67, SD = 0.69; 

• Scenario 6 (SATS, Approach with Virtual Traffic): M = 1.58, SD = 0.42; 

• Scenario 7 (Baseline, Approach to Missed Approach with Virtual Traffic): M = 2.29, SD = 0.54; 

• Scenario 8 (SATS, Approach to Missed Approach with Virtual Traffic): M = 1.67, SD = 0.44. 

A series of Wilcoxon Tests was performed to determine if EPs reported experiencing different levels of 
workload when different types of scenarios were performed using different types of procedures. Since the 
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differences between the workload ratings associated with particular pairs of scenarios were of primary interest (i.e., 
Scenarios 3 vs. 4, 5 vs. 6, and 7 vs. 8), only the results of the Wilcoxon Tests associated with these scenario pairings 
are discussed here. 


As shown by the mean workload ratings plotted in Figure 11, EPs reported experiencing higher levels of 
workload when scenarios were performed using the Baseline procedures than when scenarios were performed using 
the SATS procedures. 
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Figure 11. Mean workload ratings associated with Procedure Type x Scenario Type. 


Workload Conclusion: The Wilcoxon Tests revealed that, at a statistically significant level, EPs reported 
experiencing higher levels of workload when performing Scenario 3 as compared with Scenario 4 (p = 0.0357) and 
when performing Scenario 5 as compared with Scenario 6 (p = 0.0049). Statistically, EPs reported experiencing 
equivalent levels of workload when performing Scenario 7 as compared with Scenario 8 (p = 0.0881). In summary, 
this data illustrates that pilots reported their workload decreased when they flew SATS HVO scenarios, (surpassing 
the experimental hypothesis that “equivalent workload ratings will be associated with all scenarios”). 


C. Subjective Assessments of Situation Awareness 

EPs were administered a Situational Awareness Rating Technique (SART) instrument that included the three 
dimensions of demand, supply, and understanding as well as two independent dimensions of traffic awareness and 
navigation guidance. 13 For the dimensions of demand, supply, and understanding, EPs used a scale ranging from 1 
(low) to 7 (high) to rate each dimension. The formula shown below was used by the experimenters to calculate 
global SART ratings: 

S A = Understanding - (Demand - Supply) 

Global SART ratings can range from 1 (representing a low level of SA) to 14 (representing a high level of SA). 
In the current study, calculated SART ratings ranged from 1 to 1 1.5. For traffic awareness, scores ranging from 2 to 
7 on a scale of 1 (low) to 7 (high) were collected from the EPs, and scores ranging from 1.5 to 7 on a scale of 1 
(low) to 7 (high) were collected for navigation guidance SA. As reported below, Wilcoxon Tests were used to 
analyze the EPs’ SART scores, traffic awareness scores, and navigation guidance SA scores. Wilcoxon Tests (i.e., 
nonparametric within- subject tests appropriate for analyzing two related samples of ordinal data) were employed as 
a conservative method for analyzing SA ratings associated with discrete rating scale items. 

Procedure Type x Scenario Type 

EPs reported the following SART ratings when they performed different types of scenarios using Baseline 
procedures and SATS procedures (N=12 for each scenario): 

• Scenario 3 (Baseline, Approach without Traffic): M = 6.92, SD = 1.96; 

• Scenario 4 (SATS, Approach without Traffic): M = 8.08, SD = 1.89; 
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• Scenario 5 (Baseline, Approach with Virtual Traffic): M = 5.38, SD = 2.76; 

• Scenario 6 (SATS, Approach with Virtual Traffic): M = 7.75, SD = 1.74; 

• Scenario 7 (Baseline, Approach to Missed Approach with Virtual Traffic): M = 5.79, SD = 2.5 1 ; 

• Scenario 8 (SATS, Approach to Missed Approach with Virtual Traffic): M = 7.38, SD = 1.98. 

To determine if EPs reported different SART ratings when different types of scenarios were performed using 
different types of procedures, a series of Wilcoxon Tests was performed. Since the differences between the SART 
ratings associated with particular pairs of scenarios were of primary interest (i.e., Scenarios 3 vs. 4, 5 vs. 6, and 7 vs. 
8), only the results of the Wilcoxon Tests associated with these scenario pairings are discussed here. 

As shown by the mean SART scores plotted in Figure 12, EPs reported higher SART scores when scenarios 
were performed using the SATS procedures than when scenarios were performed using the Baseline procedures. 



Approach without Traffic Approach with Traffic Approach with Virtual 
(no holding required) (holding required) Traffic to Missed 

Approach (holding 
required) 

Procedure Type x Scenario Type 


Figure 12. Mean SART ratings associated with Procedure Type x Scenario Type. 

Situation Awareness Conclusion: The Wilcoxon Tests revealed that, at a statistically significant level, EPs 
reported higher SART scores when performing Scenario 6 as compared with Scenario 5 (p = 0.0161). Statistically, 
EPs reported equivalent SART ratings when performing Scenario 3 as compared with Scenario 4 (p = 0.2513) and 
when performing Scenario 7 as compared with Scenario 8 (p = 0.3969). In summary, the situation awareness 
hypothesis that “equivalent situation awareness ratings will be associated with all scenarios” was supported. 

D. Subjective Assessment of Traffic Awareness 

EPs reported the following traffic awareness ratings when they performed different types of scenarios using 
Baseline procedures and SATS procedures (N=12 for each scenario): 

• Scenario 3 (Baseline, Approach without Traffic): M = 5.25, SD = 1.59; 

• Scenario 4 (SATS, Approach without Traffic): M = 6.13, SD = 1.23; 

• Scenario 5 (Baseline, Approach with Virtual Traffic): M = 5.54, SD = 1.50; 

• Scenario 6 (SATS, Approach with Virtual Traffic): M = 6.46, SD = 0.40; 

• Scenario 7 (Baseline, Approach to Missed Approach with Virtual Traffic): M = 5.54, SD = 1 .42; 

• Scenario 8 (SATS, Approach to Missed Approach with Virtual Traffic): M = 6.42, SD = 0.42. 

To determine if EPs reported different traffic awareness ratings when different types of scenarios were 
performed using different types of procedures, a series of Wilcoxon Tests was performed. Since the differences 
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between the traffic awareness ratings associated with particular pairs of scenarios were of primary interest (i.e., 
Scenarios 3 vs. 4, 5 vs. 6, and 7 vs. 8), only the results of the Wilcoxon Tests associated with these scenario pairings 
are discussed here. 

As shown by the mean traffic awareness scores plotted in Figure 13, EPs reported higher traffic awareness scores 
when scenarios were performed using the SATS procedures than when scenarios were performed using the Baseline 
procedures. 



Traffic Traffic (holding Approach With Virtual 

required) Traffic (holding 

required) 

Procedure Type x Scenario Type 

Figure 13. Mean traffic awareness ratings associated with Procedure Type x Scenario Type. 

Traffic Awareness Conclusion: The Wilcoxon Tests revealed that, at a statistically significant level, EPs 
reported higher traffic awareness scores when performing Scenario 6 as compared with Scenario 5 (p = 0.0418) and 
when performing Scenario 8 as compared with Scenario 7 (p = 0.01 17). Statistically, EPs reported equivalent traffic 
awareness ratings when performing Scenario 3 as compared with Scenario 4 (p = 0.0581). In summary, the situation 
awareness hypothesis that “equivalent situation awareness ratings will be associated with all scenarios” was 
supported. 

E. Subjective Assessment of Navigation Guidance Awareness 

EPs reported the following ratings for navigation guidance SA when they performed different types of scenarios 
using Baseline procedures and SATS procedures (N=12 for each scenario): 

• Scenario 3 (Baseline, Approach without Traffic): M = 5.63, SD = 1.17; 

• Scenario 4 (SATS, Approach without Traffic): M = 6.50, SD = 0.43; 

• Scenario 5 (Baseline, Approach with Virtual Traffic): M = 5.04, SD = 1.63; 

• Scenario 6 (SATS, Approach with Virtual Traffic): M = 6.50, SD = 0.43; 

• Scenario 7 (Baseline, Approach to Missed Approach with Virtual Traffic): M = 5.21, SD = 1 .42; 

• Scenario 8 (SATS, Approach to Missed Approach with Virtual Traffic): M = 6.29, SD = 0.58. 

To determine if EPs reported different ratings for navigation guidance SA when different types of scenarios were 
performed using different types of procedures, a series of Wilcoxon Tests was performed. Since the differences 
between the ratings for navigation guidance SA associated with particular pairs of scenarios were of primary interest 
(i.e., Scenarios 3 vs. 4, 5 vs. 6, and 7 vs. 8), only the results of the Wilcoxon Tests associated with these scenario 
pairings are discussed here. 

As shown by the mean scores plotted in Figure 14, EPs reported higher ratings for navigation guidance SA when 
scenarios were performed using the SATS procedures than when scenarios were performed using the Baseline 
procedures. 
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Approach Without Approach With Virtual Approach to Missed 
Traffic Traffic (holding Approach With Virtual 

required] Traffic (holding 

required] 

Procedure Type x Scenario Type 

Figure 14. Mean ratings for SA related to navigation guidance associated with Procedure Type x Scenario 

Type. 

Navigation Guidance Awareness Conclusion: The Wilcoxon Tests revealed that, at a statistically significant 
level, EPs reported higher ratings for navigation guidance SA when performing Scenario 4 as compared with 
Scenario 3 (p = 0.0486) and when performing Scenario 6 as compared with Scenario 5 (p = 0.0098). Statistically, 
EPs reported equivalent ratings for navigation guidance SA when performing Scenario 7 as compared with Scenario 
8 (p = 0.0550). In summary, this data illustrates that pilots reported their navigation guidance awareness increased 
when they flew SATS HVO scenarios, (surpassing the situation awareness hypothesis that “equivalent situation 
awareness ratings will be associated with all scenarios”). 

V. Summary and Conclusions 

Detailed results of pilots’ flight technical error (FTE), and their subjective assessments of workload and situation 
awareness are presented and indicate that the SATS HVO concept is a viable improvement to the procedural control 
operations in use today at non-towered, non-radar controlled airfields in Instrument Meteorological Conditions 
(IMC). The HVO Flight Experiment, flown on NASA’s Cirrus SR22, used a subset of the HVO Simulation 
Experiment scenarios and evaluation pilots in order to validate the simulation experiment results. The EPs easily 
transitioned their simulator experience and familiarity with HVO procedures to the flight deck. 

Results of the subjective assessments revealed that all twelve low-time instrument-rated pilots preferred SATS 
HVO when compared to current procedural separation operations. Pilots expressed their frustration with the lengthy 
hold maneuvers on the Baseline scenarios and their relief at being able to fly the more efficient SATS approaches. 
EPs maintained airspeed and lateral path more accurately when they performed the SATS scenarios than when they 
performed the Baseline scenarios. They also maintained altitude equally well in both SATS and Baseline scenarios. 
EPs assessed their workload to be lower in SATS scenarios, and increased situation awareness with respect to traffic 
and navigation guidance. 

The piloted experiment described in this paper was a critical part of the building-block validation and 
verification process of the SATS HVO CONOPS that included multiple elements ranging from concept 
development to full-system architecture proof-of-concept demonstration that was successfully shown to the public at 
the June 2005 SATS Technical Demonstration in Danville, VA. 
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